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(g) Graphical end-user interface for intelligent assistants. 



(57) Disclosed is a graphical user interface for a 
computer system that includes one or more 
intelligent assistants. The interface includes 
composite icons comprising a graphical repre- 
sentation of a human figure, a representation of 
a desk, and a mini-icon that associates an 
assistant with the object it supports. The as- 
sistant's desk can be opened to show its con- 
tents and the human figure can move from its 
position next to the desk to a position on the 
computer display screen next to a suggestions 
dialog box that displays suggested actions to 
the user. 
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The present invention relates generally to com- 
puter system end user interfaces and more particu- 
larly to a graphical end user interface that includes in- 
telligent agents or assistants. 

In certain currently existing computer systems 5 
with graphical user interfaces, an environment is cre- 
ated that is an electronic metaphor for a workplace or 
desktop. Objects found in the electronic workplace 
are presented for users to manipulate in ways that are 
similar to the way that they are manipulated in the 10 
real workplace. Objects, such as file cabinets, fold- 
ers, documents, and printers, are displayed on the 
display screen as icons. Users manipulate these 
icons with the mouse to perform desired operations. 
For example, to file a document in a folder that is lo- 15 
cated in a file cabinet in the real workplace, the user 
opens the file cabinet, locates and opens the correct 
folder, and puts the document inside it. In the elec- 
tronic workplace, the user performs a similar proc- 
ess. The user opens the file cabinet, locates the cor- 20 
rect folder, and drops the document icon on the folder 
icon. Users have been able to use their knowledge of 
a real workplace to perform this operation. 

While the workplace or desktop metaphor pro- 
vides certain advantages to the user in ter.ms of intui- 25 
tiveness and ease of use, many have suggested that 
the next step in personal computing will move beyond 
the desktop metaphor and its arrangement of passive 
objects toward a theater metaphor of active agents. In 
the theater metaphor, work is delegated to animated 30 
intelligent objects that operate like secretaries or as- 
sistants. 

Commercialization of these ideas has just begun. 
For example, Hewlett Packard Company has intro- 
duced "agents" in its New Wave product. However, 35 
these agents are not truly intelligent Each agent is 
merely a macro-capturing facility, which records and 
plays back a sequence of commands as performed 
by the user. This ability to replay what is called a 
"script" is the lowest form of artificial intelligence. It in- 40 
eludes no ability for deductive inference or inductive 
discovery. Inthe NewWave product, the user records 
an action sequence by simply performing the action, 
but any subsequent variablization of the macro must 
be programmed by the user, if the user knows the 45 
macro language. The agent of the NewWave product 
is still passive; the script is activated only when the 
user gives it another object to work on, such as a 
spreadsheet to be formatted and printed. The next 
logical step in artificial intelligence is to add a rule- so 
based inferencing system. There has been some ac- 
tivity to build such intelligent office systems and inter- 
faces and it appears that such work has been and will 
soon be commercialized as an electronic mail func- 
tion. 55 

There is no question that rule-based agents are 
required for intelligent office systems. By definition, 
rules are required for corporate policies and regula- 



tions. The enterprise needs to define that certain ob- 
jects cannot be printed, for example. Rules are also 
the only way for the user to give instructions before 
going on vacation; there is no behavior to I earn -by- 
example because the user is not around. Simple rules 
can also be appropriate and well-formed by the user 
to handle many trivial jobs, such as automatic sorting 
of mail under appropriate key words. 

On the other hand, the difficulties of rule expres- 
sion, rule-based consistency, and a host of well 
known knowledge engineering problems suggest the 
need for case- based reasoning and an adaptive sys- 
tem that can automatically learn by example. The 
user's own behavior provides the defacto validity and 
completeness and does not require the end user's ex- 
plicit activities in knowledge engineering. Automated 
knowledge acquisition is needed for all expert sys- 
tems, but the many problems of knowledge acquisi- 
tion are exacerbated if the general office worker is 
asked also to become a knowledge engineer. 

In co-pending EU-A-No. 92480080.8, filed June 
10, 1992, entitled CO NVOLUTIONAL EXPERT NEU- 
RAL SYSTEM, assigned to the assignee of the pres- 
ent application, the disclosure of which application is 
incorporated herein, there is disclosed an adaptive 
neural system that exhibits continuous and instant 
learning, and a perfect memory. The system of EU- 
A-No. 92480080.8, has the ability to observe inputs 
and outputs, or conditions and results, and derive the 
rules that determine the outcome or result of various 
sets or combinations of inputs or conditions. Thus, the 
system of EU-A-No. 92480080.8, promises to be an 
effective engine for intelligent assistants or agents. 

The suggestion that agents or assistants will ap- 
pear as "little people" on the screen has already been 
made; however, little has been actually done to estab- 
lish the display and metaphor of its explicit interface 
architecture. The graphical portrayals in other sys- 
tems agents are still primitive. For example, the New 
Wave product uses the simple bust of a "secret 
agent", without any animation or multiple operations. 
Moreover, no system has adequately dealt with the 
problem of presenting different forms of intelligence 
within a graphical metaphor for the user. Various 
forms of intelligence are possible and should be used 
together, but this raises problems about how such dif- 
ferent forms can be represented. 

The present invention provides a method of and 
system for displaying a set of assistants and their re- 
spective objects of responsibaity together with their 
respective function and control and a method and 
system for displaying and invoking tnferenced sug- 
gestions. All assistants are collected and managed 
within a "Bullpen." The workplace or desktop contains 
an "Assistants" object or icon, which can be opened 
to show the set of assistants. Each assistant is asso- 
ciated with an object Each assistant icon includes a 
graphical human figure and a graphical representa- 



2 



3 



EP0 562 995 A2 



4 



tion of a desk that has thereon an iconic representa- 
tion of the object that the assistant serves. For exam- 
ple, in the preferred embodiment, one assistant han- 
dles documents, as shown by a document object icon 
placed on the assistant's desk. The metaphor used is 5 
that each assistant is a software robot, each with its 
own desk, and with its responsibility demonstrated by 
the object on its desk. Global properties are also in- 
cluded in the bullpen. Global properties are those 
that are not given to only one object but are displayed to 
to the separate objects in the bullpen. For example, 
in the preferred embodiment, a list of facts, such as 
the user's name, position code, vacation time, and 
the like, is displayed in a bulletin board object. Each 
assistant keeps its own set of rules, but all the assis- 15 
tants use the same set of global facts which the user 
can maintain in one place. 

In the present invention, the assistant's "learn- 
ing" is located within its head. The adaptive form of 
intelligence is provided by neural network methods, 20 
and as such is located within the assistants "brain." 
Clicking on the head of the assistant's graphical hu- 
man figure opens it to the neural representation of 
what the assistant has learned about objects of the 
particular class. The user can view the relevant van- 25 
ables for the assistant's actions and exercise "what- 
ifs." 

User developed rules for each object are located 
in the assistant's desk. Clicking on the desk opens it 
to a window that contains its contents. Rules are not 30 
learned, but are specifically stated to each assistant. 
To maintain separation of this knowledge from that 
which is learned, such rules are characterized as "in- 
structions" to the assistant from the system user and 
are kept as a document or list object within the assis- 35 
tanf s desk. Enterprise regulations are also located in 
the desk. Regulations, like user instructions, are not 
learned but are by definition simply stated as rules. 
This affords an object oriented place for a corporate 
enterprise to establish rules and an easy to use met- 40 
aphor for the user to find and reference them. Perfor- 
mance reports and management controls may also be 
located in the desk. This Is a metaphor for an employ- 
ee's performance record as a document kept in a real 
desk. The record of performance for both the neural 45 
network and the user-defined rules, such as frequen- 
cy of use and inferencial decisions versus actual out- 
come, can be accessed within this object Threshold 
setting, performance trade-offs, and the like can be 
managed from this information. 50 

Object-relevant vocabulary may also be kept 
within the desk. As voice recognition is incorporated 
into the manipulation of object-oriented interfaces, a 
proper metaphor for its use should be included. The 
idea is that the user will broadcast a command such 55 
as "Call John Adams" and the appropriate assistant 
(in this case the phone assistant) will then take the 
commanded action rather than the user saying some- 



thing like "Telephone, call John Adams." In natural in- 
teraction, people do not speak to inanimate objects. 
Vocabularies are defined within the scope of a par- 
ticular object The definition of that vocabulary, which 
are those commands that the assistant can under- 
stand and respond to, can be made here. 

In the present invention, each assistant can make 
recommendations or suggestions that can be re- 
quested from the action bar of the serviced object 
window. For example, within the context of a partic- 
ular mail document, the assistant can be called, 
whereupon it will leave its desk in the bullpen and 
then appear floating next to the opened document 
holding a list of actions. The assistant is not a normal 
object within a presentation window. Rather, it is sep- 
arate as a special type of object that can traverse the 
system or screen. In the preferred embodiment, each 
assistant can learn whether or not to display its sug- 
gestions automatically. When the user asks for sug- 
gestions from the action bar, the user is committing an 
action, like any other, that the assistant can learn to 
predict If the assistant learns the conditions under 
which the user will request its suggestions, the assis- 
tant can automatically present itself when a proper in- 
stance of the object is opened. Clicking on the assis- 
tant's feet will cancel its presence. The assistant* sfeet 
are, in effect, its "cancel" button. Clicking on its feet 
will cause the suggestion list to disappear and the as- 
sistant to return to its desk. Clicking on any item of the 
list of suggestions will cause that one suggestion to 
be selected and highlighted. In the preferred embodi- 
ment, the assistant icon has displayed adjacent to its 
head a quotation bubble with the word "Okay?" in it 
Clicking on the quote bubble will invoke all selected 
actions in the list of suggestions and the assistant will 
return to its desk. Clicking on the assistant's head will 
display its current reasoning. 

Figure 1 is a pictorial view of a computer system 
display screen with a "WORKPLACE" window display 
thereon including an Assistants icon. 

Figure 2 is a pictorial view similar to Figure 1 but 
including an "Assistants Bullpen" window. 

Figure 3 is a pictorial view similar to Figure 2 in 
which an assistant icon is enlarged. 

Figure 4 is a pictorial view similar to Figure 2 but 
including an "Assistant Desk" window. 

Figure 5 is a view similar to Figure 2 but including 
an "Inbasket" window and a "Document" window. 

Figure 6 is a pictorial view similar to Figure 5 in- 
cluding a "Suggestions" window. 

Figure 7 is a pictorial view of a a neural network 
window displayed on a computer display screen. 

Figure 8 is a block diagram representation of a 
computer system for practicing the method and sys- 
tem of the present invention. 

Figures 9-12 comprise a flow chart of a preferred 
software implementation of the method and system 
of the present invention. 
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Referring now to the drawings, and first to Figure 
1, a computer system display screen is designated 
generally by the numeral 11. Display screen 11 has 
displayed therein a window 13. Window 13 includes 
a title bar 15, an action bar 17, a client area 19, and 
scroll bars 21 and 23, all of which are enclosed in a 
window border 25. 

Title bar 15 includes, in addition to a label that 
identifies window 13 as the "Workplace" window, a 
system menu icon 27 and sizing icons 29 and 31. Sys- 
tem menu icon enables a_user to display a pull-down 
menu of actions that can be performed on the window 
itself rat her than the object or objects displayed in the 
window. For example, the user can use system menu 
icon 27 to move or close window 13. Sizing icons 29 
and 31 provide a convenient way for the user to min- 
imize or maximize, respectively, window 13. 

Action bar 17 includes a list of actions that can be 
performed on the object or objects in window 13. The 
user can move a pointer 33 with a mouse (not shown) 
to select various objects in window 13 and then in- 
voke various actions from pull-down menus that can 
be displayed beneath the items of action bar 17. Cli- 
ent area 19 of window 13 is populated with a plurality 
of objects represented by icons. In the workplace en- 
vironment, the objects are electronic metaphors for 
objectsthatauserwouldfind in a real workplace. For 
example, the workplace environment includes a cal- 
endar icon 35, which represents an electronic calen- 
dar object or application. The user can open calendar 
icon 35 and use it in ways similar to the way people 
use real calendars. Similarly, the workplace environ- 
ment includes a printer icon 37. The user can, by di- 
rect manipulation, use the mouse and pointer 33 to 
move a graphical representation of a document to 
printer icon 37 and thereby cause the document to be 
printed. 

Workplace window 13 includes, in addition to nor- 
mal office equipment objects, such as calendar 35 or 
printer 37, an assistants object represented by assis- 
tants icon 39. As will be explained in detail hereinafter, 
the system of the present invention includes a plural- 
ity of intelligent assistants that are assigned to vari- 
ous objects in the electronic workplace. The assis- 
tants have the capacity to learn rules regarding how 
various objects are to be handled, either by observa- 
tion of how the user handles the objects or by receiv- 
ing instructions or regulations from the user or the en- 
terprise for which the user works. Preferably, the as- 
sistants include neural networks of the type disclosed 
in co-pending US Patent Application Serial Number 
07/720,278, filed June 24, 1991, entitled CONVOLU- 
TIONAL EXPERT NEURAL SYSTEM, assigned to 
the assignee of the present application, the disclo- 
sure of which application is incorporated herein. 

Referring now to Figure 2, there is shown in ad- 
dition to workplace window 13, an Assistants Bullpen 
window 41 . Assistants Bullpen window 41 is simflar to 



window 13 in that it includes a title bar, an action bar, 
and scroll bars. However, the client area 43 is popu- 
lated with a plurality of assistants icons, including, for 
example, a document assistant icon, which is desig- 

5 nated generally by the numeral 45. Document assis- 
tant icon 45 includes a graphical representation of a 
human figure 47 and a mini-desk 49 which has dis- 
played thereon an iconic representation of a docu- 
ment, which identifies icon 45 as a document assis- 

10 tant. In similar fashion, each of the other assistant 
icons in Assistants Bullpen window 41 include a 
graphical representation of a human figure in a mini- 
desk with an iconic representation of the object that 
the assistant supports. Additionally, Assistants Bullp- 

15 en window 41 includes a bulletins object that is rep- 
resented by a Bulletins icon 51. Bulletins icon 51 can 
be opened to display a window into which the user 
can input facts that the assistants may need in order 
to perform their functions. For example, if the user is 

20 going to be out of the office on vacation, the user can 
identify his or her back-up and tell all assistants what 
dates the user will be out of the office. As those skil- 
led in the art recognize, windows can be opened by 
double clicking on an icon, or selecting an icon and 

25 then selecting the open action from the pull-down 
menu associated with the "File" item of action bar 1 7. 

Referring now to Figure 3, human figure 47 of 
document assistant icon 45 is shown enlarged. The 
figures of the assistant icons can be enlarged by 

30 clicking pointer 33 on the feet 53 of the figures. The 
human figures may be enlarged to provide more func- 
tionality to the interface. For example, a user might 
obtain help by clicking on the figure's shoulder and 
dismiss the help by clicking on the figure's mouth. 

35 Whenever the assistant icon is in the Assistants 
Bullpen window 41, clicking on the figures feet will 
toggle the figure between its large size and its small 
size. The metaphor is that of the assistant coming to 
attention and going back to his desk. 

40 Referring now to Figure 4, there is shown, in ad- 

dition to workplace window 1 3 and Assistants Bullpen 
window 41, an Assistant Desk window 55. Assistant 
Desk window 55 is preferably opened by double click- 
ing pointer 33 on mini-desk 49 of document assistant 

45 45. Assistant Desk window 55 includes various ob- 
jects, including instructions 57, regulations 59, per- 
formance 61, and vocabulary 63. Instructions icon 57 
represents user defined rules provided to the assis- 
tant Instructions are not learned through observa- 

50 tion, but are simply given to the assistant. For exam- 
ple, the user may instruct the assistant always to print 
correspondence from the user's immediate supervi- 
sor. 

Regulations icon 59 represents enterprise regu- 
55 lations. For example, it may be an enterprise regula- 
tion that documents carrying a certain security clas- 
sification cannot be printed on printers located in un- 
controlled spaces. Such regulations are made avail- 
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able to the assistant, perhaps as a constraint against 
any other inference. 

Performance icon 61 represents an object that 
contains a record of the performance for both the 
neural network and the user-defined rules, such as 
frequency of use and inferential decisions versus ac- 
tual outcome. The performance data provides a 
means by which the performance of the system may 
be evaluated and enhanced. Vocabulary icon 63 rep- 
resents object-relevant terms to which the assistant 
responds. As voice recognition is incorporated into 
the manipulation of object-oriented interfaces, each 
assistant will have a set of vocabulary that it can un- 
derstand and respond to. 

Referring now to Figure 5, the user has opened 
an Inbasket window 65. In-basket window 65 was pre- 
ferably opened by double-clicking on Inbasket icon 67 
of Workplace window 13. In-basket window 65 con- 
tains incoming electronic mail documents that the 
user has received. In Figure 5, the user has selected 
to view document 69 in In Basket window 65 and has 
opened a Document window 71. 

Document window 71 includes an action bar 73 
that includes an "Assistant" action item 75. In the 
present invention, whenever an object has an assis- 
tant assigned to it the system inserts an Assistant 
item in the action bar for that object's window. In Fig- 
ure 5, the user has selected Assistant item 75 and the 
system has displayed an assistant pull-down menu 
77. Assistant pull-down menu 77 includes a Sugges- 
tions item 79, as well as the items that are contained 
in the assistant's desk. Documents assistant 45 has 
"reviewed" the document of window 71 and it has, 
either by having observed the user's past actions or 
by reference to instructions or regulations, formulated 
a set of actions that the user is likely to perform on 
the document The assistant may automatically cause 
the system to perform certain actions "on its own" 
without in put from the user; however, in the preferred 
embodiment of the invention, the assistant normally 
does not cause the system to perform those actions 
without the concurrence of the user. Thus, those ac- 
tions are presented to the user as suggestions: If the 
user wishes to look at the suggested actions, the user 
may select suggestions item 79 with pointer 33 and 
thereby cause the system to display a Suggestions 
dialog box 81 , as shown in Figure 6. 

Referring to Figure 6, Suggestions dialog box 81 
is shown as if human figure 47 is holding it. In Figure 
6, human figure 47 has left its position next to mini 
desk 49 in window 41. Figure 47 has displayed next 
to its head a speech bubble 83 that has the word 
"Okay?" in it. Speech bubble 83 functions in the same 
way as the "Okay?" push-button in a normal dialog 
box. However, speech bubble 83 provides a more an- 
thropomorphic fee) to the interface. 

Suggestions dialog box 81 contains a list of sug- 
gested actions, including "callSender" and "printLo- 



cal." Thus, the document assistant has suggested 
that the user call the sender of the document and 
print the document locally. Again, the assistant waits 
for the user's concurrence before performing the ac- 

5 tion. However, in the preferred embodiment, the as- 
sistant may take certain, actions to facilitate perfor- 
mance of the action. For example, since one of the 
suggestions is callSender, the assistant may look up 
"Frank's" (the sender) telephone number and per- 

10 form other prerequisite activities to enhance system 
performance. The user can cause the system to per- 
form the suggested actions by selecting the various 
suggestions with pointer 33. If the user desires to 
have the system perform the selected suggested ac- 
ts tions, the user can do so by clicking on speech bubble 
83. If the user does not want to have the system per- 
form any of the suggested actions* the user may can- 
cer the suggestions by clicking on the feet 85 of hu- 
man figure 47. Thus, feet 85 are equivalent to the 

20 "cancer pushbutton of a normal dialog box. Sugges- 
tions dialog box 81 will remain on screen 11 until the 
user either cancels it by clicking on feet 85 or per- 
forms the selected suggested actions by clicking on 
speech button 83. When suggestions dialog box 81 is 

25 dismissed, human figure 47 returns to its position 
next to mini-desk 49 of assistants bullpen window 41 . 

If the user desires to understand how the assis- 
tant came to the suggestions of dialog box 81, the 
user may click on the head 87 of Figure 47, which will 

30 cause the system to display a neural network window 
89, as shown in Figure 7. Referring to Figure 7, neural 
network 89 shows two neurons, which indicate how 
the assistant arrived at its suggestions to call the sen- 
der and print the document locally. 

35 Referring now to Figure 8, there is shown a block 

diagram of a system according to the present inven- 
tion. The system includes a processor 91, which in- 
cludes a central processing unit (CPU) 93 and ran- 
dom access memory 95. The system may also in- 

40 elude additional memory in the form of a hard disk 
storage device 97, a floppy disk device 99, or, prefer- 
ably, both. Floppy disk device 99 is adapted to receive 
a diskette 101 that may have recorded thereon the 
software implementation of the present invention. 

45 The system of Figure 8 includes user interface hard- 
ware, including a display 103, a mouse 105, and pre- 
ferably a keyboard 107. The system may, optionally, 
include a printer 109. 

Referring now to Figures 9-12, there is shown a 

50 flowchart of a preferred software implementation of 
the present invention. Referring first to Figure 9, the 
system first begins the desktop or workplace as indi- 
cated generally at block 109. After the desktop is run- 
ning, the system monitors user input at block 111. If 

55 the system receives a user input, the system tests, at 
decision block 113 whether or not the assistants ob- 
ject has been selected. If it has, the system opens the 
assistants bullpen window at block 115 and then pro- 
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ceeds to the portion of the flowchart indicated gen- 
erally by the letter "A", which is shown in Figure 10. 

Referring to Figure 10, the system tests at deci- 
sion block 117 whether or not the bulletins object is 
selected. If it is, the system opens the bulletins win- 
dow at block 119 and returns. If the bulletins object is 
not selected at decision block 117 then the system 
tests at decision block 121 whether or not an assis- 
tants feet are selected, if the feet are selected, then 
the system tests at decision block 123 whether or not 
the assistant is enlarged. If the assistant is enlarged, 
then the human figure representation of the assistant 
is reduced at decision block 125. If the assistant is not 
enlarged, then the system enlarges the assistant fig- 
ure at block 127 and returns. Thus, in Figure 10, the 
user can toggle between a reduced assistant and an 
enlarged assistant by selecting the assistant's feet. If 
at decision block 121, the feet are not selected, then 
the system tests at decision block 1 29 whether or not 
the assistant image's head is selected. If it is, the sys- 
tem opens a neural network window at block 131 and 
returns. If the head is not selected at decision block 
129, then the system tests at decision block 133 
whether or not the mini-desk is selected. If it is not, 
then the system as indicated at "D" returns to contin- 
ue monitoring user input at block 111 of Figure 9. If, 
at decision block 133, the mini-desk is selected, the 
system opens the assistant desk window at block 1 35 
and proceeds to the portion of the flowchart indicated 
by the letter "B", which is shown in Figure 11. 

Referring to Figure 11, the system tests at deci- 
sion block 137 whether or not instructions are select- 
ed and if so, the system opens the instructions win- 
dow at block 139 and returns. If instructions are not 
selected, the system tests at decision block 141 
whether or not regulations are selected. If they are, 
then the system opens a regulations window at block 
143 and returns. If regulations are not selected, the 
system tests at decision block 145 whether or not per- 
formance is selected. If it is, the system opens a per- 
formance window at block 147 and returns. If perfor- 
mance is not selected, then the system tests at deci- 
sion block 149 whether or not vocabulary is selected. 
If it is, then the system opens the vocabulary window 
at block 151 and returns. Finally, if vocabulary is not 
selected, then the system returns as indicated at "D" 
to continue to monitor user input at block 111 at Figure 
9. 

Referring again to Figure 9, if the user input is not 
a selection of assistants, the system tests at decision 
block 153 whether or not an other object is selected. 
If it is, the system opens the window for the other ob- 
ject at block 155. Then, the system tests at decision 
block 1 57 whether or not the selected object is assist- 
ed. If it is not, the system returns to block 11 to con- 
tinue monitoring user input If the object is assisted, 
then, at block 159, the system Includes an assistant 
item in the action barfor the object window. Then, the 



system tests at decision block 161 whether or not the 
user asks for suggestions in such cases. If the assis- 
tant has learned that the user does ask for sugges- 
tions in such cases, the system automatically opens 
5 the suggestions dialog box at block 163. If the user 
does not normally ask for suggestions in such cases, 
then the system tests at decision block 165 whether 
or not the assistant item is selected from the action 
bar. If it is not, the system returns to block 111 to con- 
to tinue monitoring user input If the assistant item is se- 
lected, then the system displays the assistant pull- 
down menu at block 167 and waits for user input If, 
at decision block 169, the user selects suggestions, 
then the system opens the suggestions dialog box at 

15 1 block 163. If the user does not select suggestions, 
then the system goes through the routine indicated 
generally by "B", which is shown in and described in 
connection with Figure 11. 

Referring still to Figure 9, after the system has 

20 opened the suggestions dialog box at block 163, the 
system proceeds as indicated generally by the letter 
"C", which is shown in Figure 12. Referring to Figure 
12, the system displays any suggestions in the sug- 
gestions dialog box at block 165. Then, the system 

25 tests at decision block 167 whether or not the assis- 
tant bullpen window is open. If it is, the assistant icon 
is erased from the bullpen window at block 169. After 
the assistant icon has been erased from the bullpen 
window, or if the bullpen window is not open, the sys- 

30 tern displays the assistant icon next to the dialog box 
at block 171 and displays the "Okay?" speech bubble 
next to the assistant icon at block 173 and monitors 
user input. If, at decision block 175, the user input is 
the selection of a suggestion, the selected suggestion 

35 is highlighted at block 177 and the system returns to 
continue monitoring user input. If, at decision block 
179, the user selects the head of the assistant icon, 
the system opens the neural network window at block 
181 and returns to continue monitoring user input. If, 

40 at decision block 183, the user selects the "Okay?" 
speech bubble, the system performs all selected sug- 
gestions at block 185. If the user input at decision 
block 183 is not selection of the "Okay?" speech bub- 
ble, then the system tests at decision block 187 

45 whether or not the assistant's feet are selected. If 
they are not the system returns to continue monitor- 
ing user input. If the feet are selected, at decision 
block 187 or if okay has been selected, as described 
above, the system closes the dialog box and erases 

so the assistant icon and "Okay?" speech bubble at 
block 189. Then, the system tests at decision block 
191 whether or not the assistant bullpen window is 
open. If it is, the system displays the assistant figure 
icon in the bullpen window at block 193 and returns 

55 to block 111 of Figure 9 to continue monitoring user in- 
put Finally, referring to Figure 9, if the user input is 
something other than selection of the assistants icon 
or another object the system processes such other 

6 
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user input at block 195 and continues to monitor user 
input at block 111. 



Claims 

1. A method of performing an action in a computer 
system that includes at least one intelligent assis- 
tant and a display screen, which comprises the 
steps of: 

displaying an object window on said display 
screen, said object window including an action 
bar; and 

displaying an assistant item in said action bar 
whenever the object of said application window 
has an intelligent assistant associated therewith. 

2. The method as claimed in claim 1 , including the 
steps of: 

selecting said assistant item; and 
automatically displaying a pull-down menu con- 
taining a suggestions item. 

3. The method as claimed in claim 2, including the 
steps of: 

selecting said suggestions item; and 
automatically opening a suggestions dialog box 
in response to selecting said suggestions item, 
said suggestions dialog box having displayed 
therein at least one suggested action. 

4. The method as claimed in claim 3, including the 
steps of: 

selecting, an action from said suggested actions; 
and automatically performing said selected sug- 
gested action. 

5. The method as claimed in claim 4, including: 
displaying a graphical representation of a human 
figure, adjacent to said suggestions dialog box, 
and a speech bubble adjacent to the head of said 
human figure, wherein said step of selecting an 
action from said suggested action includes se- 
lecting said speech bubble. 

6. The method as claimed in claim 5, including: 
selecting the head of said human figure; 
automatically opening a neural network window 
in response to selecting said head; and 
displaying in said neural network window a 
graphical representation of a neural network as- 
sociated with said intelligent assistant 

7. The method as claimed in claim 5, including: 
selecting the feet of said human figure; and 
closing said suggestions window and removing 
from said screen said human figure in response 
to selecting said feet 



8. The method as claimed in claim 1, including the 
step of automatically displaying a suggestions di- 
alog box whenever said intelligent assistant has 
learned that the user of said system typically re- 

5 quests suggestions. 

9. A method of performing an action in a computer 
system that includes at least one intelligent assis- 
tant and a display screen, which comprises the 

10 steps performed by said computer system of: 

displaying an assistants icon on said display 
screen; and 

displaying an assistants window in response to 
user selection of said assistants icon. 

15 

10. The method as claimed in claim 9, including the 
steps of: 

displaying at least one assistant icon in said as- 
sistants window, said assistant icon representing 
20 said at least one intelligent assistant. 

11. The method as claimed in claim 10, wherein said 
assistant icon includes: 

a graphical representation of a human figure; 
25 a graphical representation of a desk positioned 

adjacent to said human figure; and 
a mini-icon representative of an object associated 
with said assistant. 

30 12. The method as claimed in claim 11, including the 
steps of: 

opening an assistant desk window in response to 
user selection of said graphical representation of 
a desk. 13. The method as claimed in claim 11, 
35 including the step of: 

replacing said graphical representation of a hu- 
man figure with an enlarged graphical human fig- 
ure in response to user selection of the feet of 
said graphical representation of a human figure. 

40 

13. The method as claimed in claim 13, including the 
step of: 

replacing said enlarged graphical human figure 
with said graphical representation of a human f ig- 
45 ure in response to user selection of the feet of 
said enlarged graphical human figure. 

14. The method as claimed in claim 11, wherein said 
at least one intelligent assistant includes a neural 

so network and including the step of opening a neu- 

ral network window in response to user selection 
of the head of said graphical representation of a 
human figure. 

55 15. The method as claimed in claim 11 , including the 
steps of: 

opening an object window for said object associ- 
ated with said at least one intelligent assistant; 
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displaying a suggestions dialog box in associa- 
tion with said object window; and 
moving said graphical representation of a human 
figure from said position adjacent to said graphi- 
cal representation of a desk to a position. adjacent 5 
to said suggestions dialog box. 

16. The method as claimed in claim 16, including the 
steps of: 

closing said suggestions dialog box and moving 10 
said graphical representation of a human figure 
back to said position adjacent to said graphical 
representation of a desk, all in response to user 
selection of the feet of said graphical representa- 
tion of a human figure. 15 
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